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(54) Packet network transit delay measurement system 



(57) A system for analyzing and estimating average 
transit delay and high percentiles of delay in a packet 
network (10) uses detailed packet switch measure- 
ments (30) that are normally stored in a network data- 
base for maintenance and engineering purposes. The 
system randomly selects source (12) and destination 
(14) points and samples the stored maintenance meas- 
urements (30) for network circuit elements located be- 
tween the randomly selected source and destination 



points. These sampled measurements are transmitted 
to the system processor via electronic mail and the sys- 
tem generates a single server queuing model (40) to de- 
termine average and variance of delay for each compo- 
nent. A network-of -queues model is then constructed to 
determine the end-to-end average delay and end-to- 
end variance of delay. An Erlang probability distribution 
is then applied to these end-to-end values to provide 
statistics of packet transit delay in the packet network 
(10). 



FIG. 2 




Printed by Jouve. 7500 1 PARIS (PR) 



EP 0 716 525 A2 



Description 



BACKGROUND OF THE INVENTION . . 

5 Field of the Invention . .. . 

The.present invention, relates to. data cpmmunictions networks, and more particularly to a system for measuring 

transit delay in a packet network. 

..10 Description of the Related Art 

In many data communications networks" data is generated in high intensity bursts that are often separated by idle 
periods therebetween. The average data rate may, therefore, be much lower than the peak rate. An economic way of 
Transmitting this type of intermittent (bursty) data is to assemble the data into what is known in the. art as packets and 
is interspersing packets from several different channels on a single, physical communication path. Th.s commun.cat.on - 
path is therefore occupied by. a. particular packet only for. the time corresponding to the packet length and the com- 
munication path is then available for use by other packets that may be between different data terminals. 

A packet refers to a group of bits that are switched in the network as an integral unit. A packet may also .nclude 
. a header that is used to identify, that packet. A.packet is formatted in a particular way and may conta.n data, and 
20 destination, origination and control. information. .. . . . . . • 

in a typical packet-switching network, a number of users can be interconnected. Packet switches are , ut.hzec ,n 
the network to sort packets received from one circuit and direct these packets to another circuit accord.ng to the header 
information for each packet. Atypical packet switch is AT&T's 1 PSS™ packet switch wh.ch .mplements the CCITT X. 
25 packet-switching protocol. , , ^ - , , - 

25 m any data communications network, a primary concern in the, performance of the network .s the time m which 

information is processed and transmitted back to the user. This time, is typically referred to as response time or network 
transit detay Network transit delay in a packet-switching network may be a function of data transmiss.on speed, prop- 
agation time, distance through the network, packet-switch, processing time and queu.ng .delay 

To evaluate the performance of a packet network,, transit delay » generally measured from the time ttie las lb t of 
30 a particular packet enters the providers originating central office equipment (e.g.,. ong.nat.ng P^wrtch) to the me 
the first bit of the packet leaves the provider's destination central.office equipment (e.g.. destination packet switch). 

In designing and maintaining data communications systems, it is essential to m.n.m.ze network trans [delays. In 
MmSZSZSS* today have strict requirements regarding the permissible transit delay in the network Customers 
wm ofte" requ^the provider of the data communications network .to , pay a penalty if the trans* delay exceeds a 

35 Pre 1SoS important for the designed maintenance of.a data communications system to ef.ectK,e ly = 
transit delay in the packet network. Moreover, customers of the system frequently require periodic measurements of 
ans detev o ascertain whether the system is performing within, the limits of the customers' service contracts 
; • ti le ^Salmber of measurement systems, used today to estimate or measure network trans,t delay 

40 These measurement systems generally fall, into two categories .and are illustrated in FIG. t. ^ 

The St tvpTof measurement system, is a protoco. analyzer 20 which includes hardware connected o the packet 
netwo l ior%Z .and measuring transit.delay. Transit delay is measured by connecting a ^ ^20 
■ iTa dVsired measurement point in the network^ and essentially timestamping a.part,cu,ar packet to determine the 

*^Zt^^^™^ - ^ extremely expensive 
20 aleach'deld measurement point. Thus,, if a customer ^^.•fi,^ 
often do orotocol analyzers 20 .would be required at each point of access into the network 10. 

I adSonTo fhe expense associated with requiring a protoco. analyzer 20 at each desired measurement po j* 
each protocol analyzer 20 must be. .ocated.in .the. service node (i.e, the packet sw.tch) in 

requires space for locating the protocol analyzer 20, power for operating the equipment, and the ab.My to access the 

'^It? therefore desirable to provide a cost effective, non-invasive measurement system to estimate transit delay 
in a packet ^SSt^S doJnot require hardware or measuring software to be added to the network 10. 
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SUMMARY OF INVENTION 

The present invention provides a cost effective system for estimating the end to-erid packet delay distributions in 
a packet network. The system analyzes and estimates average transit delay and high percentiles of delay in a packet 

s network by utilizing detailed packet switch measurements that are normally collected and stored in a network database 
for maintenance and engineering purposes. The system randomly selects various source and destination points and 
samples the stored maintenance measuremeritsfor specific network circuit elements between the randomly selected 
source and destination points. These sampled measurements are transmitted to the system processor and a single 
server queuing model is generated to determine average and variance of delay for each identified element. A network- 

io of-queues model is then constructed to determine the end-to-end average delay and ehd-to-end variance of- delay. An 
Erlang probability distribution is then applied to these end-to-end values to provide statistics of packet transit delay in 
the packet network. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a network overview illustrating the prior art methods of measuring packet network transit delay. 
FIG. 2 is an illustration of the measurement/ modelling approach of the present invention. 
FIG. 3 is an illustration of a preferred embodiment' of the present invention; 

' FIG. 4 illustrates the network-of-queues model utilized in thepreferred embodiment of the present invention. 

20 FIGS. 5-7 are block diagrams of steps performed in the preferred embodiment of the present invention. 

. •- • . . ■ \ .< ■■• , : , , . ■ . i .. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The present invention provides a cost effective method for estimating the end-to-end delay distributions in a packet 
25 network. Referring to FIG. 2, a packet network 10 is generally shown connecting a data source 12 to a data destination 
14. Although not illustrated, it is commonly known that a packet network 10 may consist of numerous elements or 
components, including, but not limited to, access lines,' gateway links, inter-switch trunks, packet switches; packet 
assembler and disassemblers (PAD), concentrators,' multiplexers and interfaces. ' " 

As part of the normal operation'of a p'acket network '10, it is common for a provider of the network 10 to collect 
30 certain packet switch maintenance measurements or data 30 for engineering and maintenance purposes to keep the 
network 10 operational. The maintenance measurements 30 are typically stored in a database 32 of network 10 as 
illustrated in FIGS. 2/arid 3. The maintenance measurements 30 also represent the amount of packets that are trans- 
mitted through the network 10. " ^ ' '[' 

For each circuit in the network 10," these maintenance measurements 30 commonly include packet counts, byte 
35 counts, speed of circuit, errored frame rate, average packet size, packet retransmission rate, packet switch utilization 
and circuit utilization. Although' well known in the art, the above-mentioned maintenance measurements 30 will be 
' generally described below ' ' ' ' ' 

A packet count generally refers to the numberof packets switched by a particular element (i.e., a packet switch) 
in the network 10 in a given period of time/ Byte cduht refers to the number of bytes contained in the information field 
40 of the packets that have been switched by the particular element in the network 10 in a specified interval of time. 

Speed of arcuit is a fixed parameter that generally refers to the capacity of the network circuit in bits per second. 
Errored frame rate refers to the number of frarVies (data link layer) that were received in error in a given interval of time. 

" Average packet'size generally' refers to the average 'size of packets measured in bytes. Packet retransmission 
rate refers to the numberof packets retransmitted (e.g., transmitted more than once) in the network 10 in a given period 
45 " of time. * • " " ' ' v '* * *' " ...... . . 

Packet switch utilization refers to the fraction of the "capacity of the'* packet switch processor (not shown) that is 
utilized in a given interval of time. Packet switch utilization is based upon" the previously described packet count. 

Circuit utilization refers to the fraction of the capacity of thenetwork circuit (riot shown) that is utilized in a specified ' 
' period of time. Circuit' Utilization is derived from the previously described byte count. 
50 As is illustrated in FIGS.' 2 and 3, the present invention utilizes the above-mentioned maintenance measurements 

30 for specific network elements to generate a model 40 of the network' 1 0. The network model 40 is utilized to estimate 
the end-to-end delay in the network 10. 

( The sequence of operation of the preferred embodiment of the present invention is illustrated in FIGS. 5-7. The 
maintenance measurement's 30 in network I0~are collected (step 100) and stored (step 110) in database 32. As pre- 
55 ' viously stated, steps 100 and 110 are commohly performed in packet networks 10. 

Since a typical packet network 10 may be connected to multiple source points 12 and destination points 14, it is 
necessary to identify a particular" source point 12, destination point 14, date and time of measurement in order to 
determine the packet network transit delay (step 120). This identification* may be accomplished by user input. 
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In the preferred embodiment of the present invention, however it is preferable to randomly sample different points 
in the network 10. The preferred embodiment, therefore, randomly selects a desired source point 12 and destination 
point 14 for measuring the network transit delay therebetween, and randomly chooses the date and time of such 

measurement. - ,.- ( '/'""• .. . '* " ' ... 

5 By randomly selecting the source and destination points 12,14, and date and time of sampling, the present inven- 

tion is capable of generating a model 40 of the network 10 and need not take actual measurements between every 
source and destination point 12, 14 connected to the network 10. This is desirable because, for every N end points in 
network 10, there exist N 2 end-to-end points. Thus, if there were 5000 end points in network 10 : there would be 25 
million paths to measure. Assuming that a customer desired hourly network transit delay measurements (also referred 

io: . to as quality of service (QOS) measurements), thiswould amount to approximately 20 billion QOS measurements per 
month. Selecting random end-to-end paths based upon randomly selected source and destination points 12, 14 and 
taking measurements at random times dramatically decreases the number of QOS measurements necessary to gen- 
. erate the model 40 of the network 10. " \ • 

Although the preferred embodiment- of the present invention samples the packet network transit; delay between 

'is randomly selected source and destination points 1 2, 14, it is understood that this method is utilized to make the system 
more practical. However, it is not a prerequisite of the present invention to randomly sample network transit delay 
While it may be unpractical or inefficient, the present invention may also be utilized to actually measure transit delay 
based upon each source and destination point" 12, 14 connected to network 10. 

Once the desired source and destination points 12, 14 are identified in step 120, the system identifies the packet 

20 switches in network 10 that each end point is connected to (step 130). The network packet switches are determined 
based upon the particular source and destination points 12, 14 previously identified in step 120. 

In step 140, the system also identifies the trunks and tandem switches (if any) between the previously identified 
source and destination points 12. 14. Having determined the desired source and destination points 12, 14 in step 130, 
the present invention utilizes known routing tables in step 1 40 to identify the trunks and tandem switches in the network 

25 10 between the source and destination points 12, '14. Thus, for each desired source and destination point 12, 14, the 
system identifies the trunks and associated packet switches in network 10 connecting the desired source and desti- 
nation points 12, 14. 

Having identified the associated packet. switches in network 10 that connect the desired source point 12 to the 
desired destination point 1 4, the system .then reads (step 1 50) the relevant maintenance measurements 30 collected 

30 and stored in the network database 32 (steps 1 00, 1 1 0) for each identified packet switch corresponding to the identified 
source and destination points 12, 14, transports (step J 60) the relevant maintenance measurements 30 to the modelling 
processor 50 and feeds (step 170) the relevant maintenance measurements 30 into the network modelling program 
operating in the modelling processor 50 as illustrated in FIG. .3. Methods well known ; in the art to transfer- such data 
include but are not limited to, electronic mail systems or file transfer protocol. . . 

3S By utilizing the relevant maintenance measurements 30, the.modelling, processor 50 estimates end,to-end delay 

distributions for packet network 10. The system generates a model 40 of the network 10 by generating -single server 
M/G/1 queuing models for individual network elements^.g., identified packet switches) and constructing a network- 
of-queues model in a series as illustrated in FIG. 4. For analytical tractabil.ty all queues are assumed to be independent. 
As will be discussed below, an Eriang probability distribution is then fit to two end-to-end delay parameters (average 

40 and variance of delay) to predict service delay performance, of the network 10. . , 

Thus for each network element (N), the system generates a network element single server queuing model. Each 
element model has two primary output parameters: average transit delay (rn s > and variance of packet delay (a,) The 
average transit delay (m,) is broken down into two components, namely, a constant (non-variable) component (m, ) 
and a variable component (m*>. The variable component (m,*) is derived from the, randomly variable portion of the 

packe^delay^ ^ ^ ^ ^ ^ ^ the average transit delay (m^for each network,element (N) is calculated using the 
following well known formula for an M/G/1 queuing model: . 

: •; - m < ,= x <- (1 + p w^) ] 

where x s represents average service time, Pi server utilization, a xj 2 variance of service time, and C bi 2 =a xi 2 /(X|) 2 (the 

Refe^ 1 90 in FIG. 6, the variance of packet delay (o?) is determined using the following well known 

formula for an M/G/1 queuing model: 



so 
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2(1-Pi) 



Pi *i 



where x 3 represents the third moment of service time, ij average service time, pj server utilization,' a xi 2 variance of 
service time, and C bi 2 =a xj 2 /(Xi) 2 (the coefficient of variation)'. 

Once the average transit delay (nrv,) and variance of packet delay (o^ 2 ) are determined for each network element 
10 (N) in steps 180 and 190, end-to'-end performance measures for network 10 are obtained in step 200 by addition of 
each and o^ 2 resulting in end-to-end average delays (both fixed (m<=) and variable (m v )) and end-to-end variance 
of delay (a 2 ). - - 

The details of step 200 are given below: Let m } and Cj 2 denote the average and variance of delay of the ith network 
element (N). Let mf and nry denote the constant and variable portions of the average delay, respectively. Accordingly, 
is the end-to-end average (m) and variance (a 2 ) packet delay for the network 10 is determined as follows: 
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c. 



1-1 
N 



2. 



Once step 200 is completed and the'end-to-end average packet delay (m) and variance of delay (a?) are determined 
for network 10. a simple Erlang probability distribution is applied to the end : to-end average and variance of delay in 
step 210.. thereby providing the high percentiles of delay (e'g., 95th percentile) frequently required by customers. Only 
the variable portion of the average delay (m*) and the variance of delay (a 2 ) are used in the Erlang distribution. The 
fixed portion of the average delay m c is excluded to avoid biasing the fit through effects of propagation delay and 
constant components of processing or insertion delays. The bias tends to make the Erlang distribution have high order, 
which may cause under-estimation of the high percentile of delay. 

The Erlang model requires two parameters; namely, average and order (r): The average is simplyset to m v , while 
the order (r)=min[max(1 ,L (m v /o) 2 J]; 1 0]. The order (r) is limited to values between 1 and 10. The value of 1 corresponds 
to an M/M/1 queue, while the' maximum value of 10 ensures that a certain minimum variability is maintained in the 
model 40.' * : •' *v ,., 

The resulting Erlang distribution as a function of delay (d) is denoted E(m v : r,d). The pth percentile of delay corre- 
sponding to a given probability (p) is denoted E" r (m v ,r,p): The- network delay performance measures arethen simply 
estimated as follows: ' - • ' ' " 
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Average Delay: m " J .' 1 • c ' ■■ ■ * - - - ! 

pth Percentile of Delay: E* 1 (m v ;r 5 p)+m c : 1 ' : * 

The table below shows the' relationship among E*H™ v Vr,p), i andp for various example values of r and p. Note that 
for given values of r and p, E^mV.p) is proportional to m v . - 



r 


p=0.95 


: p=0.90 


95th %tile/m v 


\ 90th %tile/m v 


1 


3.0 


2.4 


2 


2.4 


195 


3 


2.1 


1.8 
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1.95 


1.7 
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(continued) 



5 





p=0.95 


P = 0.90,.: r . 


'95th %tile/m v 


90th"%tiie/m v 


5 ... 


.1.85 


.1.60 


6 


1.8 . 


1.55 


7 

1 : 


1.7 


1.55 


8 


1.65 


1.5 


9 


1.65 


1.45 


10 


1.60 


, 1.45- 



Once the high percentiles of delay are determined in step 210, the system outputs the results as illustrated in step 
220 of FIG. 7. In the event that all data has not been read (step"230) : the system repeats steps 150 through 220 until 
all of the desired data has been processed. When all of the data has been read in step 230, the system generates a 
complete QOS summary report (step 240) detailing the performance of the network 10 with respect to its packet transit 
delay objective. 

It is to be understood that the above description is only of a preferred embodiment of the present invention. Nu- 
merous other arrangements may be devised by one skilled in the art without departing from the spirit and scope of the 
invention. The invention is thus limited only as defined in the accompanying claims. 
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Claims 



1 . A packet network transit delay measuring system for estimating packet transit delay in an existing packet network 
having a plurality of packet switches, comprising the steps of: 

(a) collecting packet switch measurements; 

(b) determining element packet delay statistics based upon said collected packet switch measurements; 

(c) determining end-to-end packet delay statistics from said element packet delay statistics; 

(d) randomly selecting a source point and a destination point to sample packet transit delay therebetween; 

(e) identifying the packet switches located on a circuit between the randomly selected source and destination 
points; and 

wherein the element packet delay statistics are determiried for each identified packet switch based upon col- 
lected measurements for that identified packet switch. 

2. A system as claimed in claim 1 , further comprising the step of storing the collected packet switch measurements 
in a network database. 

3. A system as claimed in claim 2, wherein said collected packet switch measurements are transmitted from said 
network database to a processor means for determining element and end-to-end packet delay statistics. 

45 4. A system as claimed in claim 1 , wherein the packet switches located on the circuit between the randomly selected 
source and destination points are identified using routing tables. 

5. A system as claimed in claim 3, wherein said collected packet switch measurements are transmitted from said 
network database to said processor means by electronic mail. 



6. A system as claimed in claim 3, wherein said collected packet switch measurements are transmitted from said 
network database to said processor means by file transfer protocol. 

7. A system as claimed in claim 1 . wherein the determination of said element delay statistics for each identified packet 
switch comprise the steps of: 



(a) 



determining average packet transit delay based upon said packet switch measurements: and 
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(b) determining variance of packet delay based upon said packet switch measurements. 

8. A system as claimed in claim ?! wherein the determination of said' end-to-end delay statistics comprise the step 
of summing the average packet transit delay and variance of : packet delay for each identified packet switch. 

9. A system as claimed in claim 8, wherein high percentiles of delay statistics are determined by applying an Erlang 
probability distribution to the end-to-end delay statistics. 

10. A system for estimating packet transit delay in an existing packet network having a plurality of packet switches 
said system comprising: : 

(a) means for collecting and storing maintenance measurements for said packet switches; 

(b) processing means for determiningelement and end-to-end packet delay statistics based upon said [packet 
switch] collected and stored maintenance measurements; and 

(c) means for transmitting said stored maintenance measurements from said collecting and storing means to 
, said .processor: and wherein said processing means; , . 

(d) randomly selects a source point and a destination point to sample packet transit delay therebetween: and 
(ej identifies the packet switches located on a circuit .between the randomly selected source and- destination 
points. 

11. A system as claimed in claim. 10, wherein the element packet delay statistics are determined for each identified 
packet switch based upon collected measurements for that particular identified packet switch. ... 

12. A system as claimed in claim 10, wherein the packet switches are identified using routing tables. 

13. A system as claimed in claim 12, wherein said element delay statistics comprise: 

(a) average packet transit delay for each identified packet switch determined from the transmitted measure-, 
ments for each particular identified packet switch: and 

(b) variance of packet delay for each identified packet switch determined from the transmitted measurements 
for each particular identified packet switch, . " -t . 

14. . A system as claimed in claim 1.3, wherein said processor determines said end-to-end delay statistics by summing 
, the average packet transit delay and variance of packet delay for each identified packet switch: 

15. A system as claimed in. claim 1 4, wherein said processor determines high percentiles of delay statistics by applying 
an Erlang probability distribution to the end-tp-end delay statistics. 
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FIG. 2 
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FIG. 5 
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FIG. 6 
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